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How can school officials manage their 
responsibilities in ways that are likely to improve the quality of 
the product they provide? There is a logical sequence of activities. 
First, school officials must be able to diagnose where problems 
exist. Then they must learn to apply treatments systematically, to 
assess carefully the success or failure of the treatments applied,, 
and last, but by no means least, to alter the treatment in accordance 
with the results of the assessment. No school system does all these 
things at present. Instead school officials are loathe to make 
decisions, and in those instances where decisions are made, they are 
even more reluctant to alter a chosen course of action based on the 
results of their efforts. These statements reflect three years of 
working closely with a large urban school system, that of Atlanta, 
Georgia. The bulk of this discussion is divided into five parts. The 
first section sets out the assumptions which shaped our work in 
Atlanta. The next two sections explain the technique developed for 
signaling extremes of relative performance and the uses and 
limitations of that kind of information. The final two sections 
describe the impact of information about relative performance on 
selected activities of Atlanta administrators. (Author/JM) 
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PERFORMANCE INFORMATION: ONE SCHOOL SYSTEM'S RESPONSE 
I, Introduction 

Public education, especially urban education, is big business. The 
operating budget of each of the 20 largest school systems exceeds $100 
million annually. Gone a"*"e the days, of the one room school-house and 
the small, friendly school district where each administrator could right- 
fully claim to make decisions on the basis of what was "best" for little 
Johnny Jones, because the administrator actually knew Johnny and his 
problems. Today, administrators in big urban school systems regularly 
make decisions which affect thousands of Johnny Joneses in dozens of 
schools. And in education today, the consumers — students, parents, 
teachers, the community at large — are far from satisfied with the decisions 

being made. In short, public education is big business and it's in 

trouble. 

Like all business, the major challenge facing education is to 
continually improve the quality of its product. But other businesses 
judge their success by public acceptance of their products, translated 

into a single simple standard profit. Not so public education. It 

has a virtual monopoly on the provision of education and no profit motive 
against which to gauge its success. More importantly, there is no 
agreeraent on how the success of public education should be measured or on 
how to interpret and use the imperfect indicators V7e do have. 

Consider an exanple from the world of business. A manufacturer 
produces a 5-speed bicycle to sell for $99 and expects to make a $10 
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profit on each sale. Everyone agrees how a bike should perform and, so, 
the manufacturer knows immediately what to do to correct any problems 
with his product. If the bike won't stop, work on the brakes; if the 
handlebars are loose, tighten them. The diagnosis and treatment are 
the same, whether the bike is hand-made or produced on an assembly-line. 
The symptoms tell the manufacturer precisely v;here to look for the trouble 
and what to do about it. And of course, if the public won't buy the 
bicycle at all or the manufacturer can't produce his product and still 
return a profit, then he will go out of business. 

But education is not that simple. First there is no agreement on 
how a good public school system should perform. There is no universal 
product in education: children differ and their educational needs differ. 
Consequently the proper diagnosis and treatment of one child's educational 
problems will not automatically succeed when applied to other children 
throughout a school system. Moreover, in education, unlike business, the 
relationship between the process and the end product of the effort is not 
completely understood. If a child can't read, does the fault lie with 
the child, his home situation, his health, his teacher, his textbooks and 
materials — or with a dozen other factors which may contribute to the 
problem? And finally, when all elsie fails, public education continues. 
As a '^public good," attendance at school is conpulsary for about 10 years. 
The customer cannot simply refuse to buy the product. And for most 
customers, there is no practical alternative to public education. 

•So, how should administrators of large school systems behave? How 
can school officials manage their responsibilities In ways that are likely 
to improve the quality of the product they provide? Vfriat treatment will 
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work — in which situations and for which students? There are no pat 
answers to these questions. But, I submit that there is a logical sequence 
of activities which taken together should improve the quality of public 
education. First, school officials must be able to diagnose reliably and 
accurately where problems exist. Then they must learn to apply treatments 
systematically, to assess carefully the success or failure of the treat- 
ments applied, and last, but by no means least, to alter the treatment in 
accordance with the results of the assessment. No school system does all 
of these things at present. Instead, school officials are loathe to make 
decisions, and in those instances where decisions are made, they are even 
more reluctant to alter a chosen course of action based on information 
about the results of their efforts. 

These statements reflect three years of working closely with a large 
urban school system in an attempt to improve the system's capacity to plan 
and manage its activities. The school system was Atlanta, Georgia. At 
first, my colleagues and I at The Urban Institute concentrated on developing 
a technique which would locate examples "of the relative success and failure 
of similar schools in teaching basic f. kills. Once the technique was 
developed, we, as researchers, stepped back and monitored the actions of 
school officials to see how, if at all, they would use hitherto unavailable 
information to make decisions about the day-to-day operations of the school 
system. 

The remainder of this presentation is divided into five parts. The 
first section sets out the assumptions which shaped our work in Atlanta. 
The next two sections explain the technique we developed for signaling 
extremes of relative performance and the uses and limitations of that 
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kind of information. The final two sections describe the impact of 
information about relative performance on selected activities of Atlanta 
administrators. 



II. Assumptions 

We began our work in Atlanta with a set of assumptions about the 
operations of any large, urban school district: 

1. That school system administrators (from principals on up) are 
managers who make decisions involving the allocation of resources 
expended on the educational process (resources in the sense of 
dollars, allocation of instructional materials, teaching staff, 
time of special staff, etc.). 

2. That most large school systems already have an abundance of data 
about students and their performance which can be organized in 
new ways to provide concise, easy to- understand information 
useful for managing the day-to-day operations of the schools. 

3. That these administrators, while surrounded by data, make 
decisions on the basis of little or no current information about 
even those educational outcomes that can be measured. In fact, 
administrators usually don't know where to look for current 
examples of the school system's success or failure in teacing 
basic skills. 

4. That management information sfiould be based on performance and 
should use the school and the grade within a school — not the 
individual student — as the basic reporting unit, because 
decisions about how to allocate educational resources inevitably 
involve the school or the grade within the school as their lowest 
common denominator. 

5. That management information which tells administrators where 
to look for current examples of success or failure should take 
into account the high correlation between student achievement 
in school and socio-economic background. Therefore, any system 
which identifies success or failure of the school system to 
teach basic skills should center on relative rather than 
absolute performance. 

6. That if administrators had current, easily understood information 
about V7here to find extremes of relative performance, they would 
make "better" decisions — that is, they would begin to target 
more precisely the limited resources available to improve the 
quality of public education. 
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These assumptions guided the development of information for Atlanta 
administrators which enabled them to locate specific grades in Individual 
elementary schools in which the level of average student performance was 
significantly better or worse than average student performance in schools 
of similar economic level. 

III. Signals of Relative School Performance 

The approach we used in Atlanta compared performance among schools 
serving similar students, identified significant cases of extreme per- 
formance, and displayed the results as a series of charts in which red 
"signals" denote levels of relatively low performance and blue "signals" 
denote> levels of relatively high performance. .In this project, we used 
mean (average) achievement on the annual standardized tests as the 
measure of performance. Schools were identified as similar based on the 
level of student participation in the free and reduced-priced lunch program. 
Since entry Into the free and reduced price lunch program is determined 
by family size and income, the percent o'f students who participated in this 
program provided an indicator of the percentage of poor students at each 
school. [This variable alone accounted for 50 to 80 percent of the 
variation in average scores in each grade level, even though the variable 
represented the average participation of an entire school rather than an 
individual grade.] 

To produce signals of relative performance for one grade, average 
scores on the reading and the arithmetic subtests for a given year were 
plotted against the amount of participation in the free and reduced-^price 
lunch program for eacTi school. The result of this effort was one ' 



ERLC 



6 



scattergram (like the one shown in the Figure A) for each grade and each 
subtest (7 grades x 2 subtests = 14 scattergrams for each year). The 
amount of participation in the subsidized lunch program, is shown along 
the horizontal axis; mean achievement, along the vertical axis. Atlanta 
schools at the end of the scattergram, labelled "High" participation, 
enrolled more students from poorer economic backgrounds. Each mark (+) 
on the scattergram represents the 5th grade average for all students who 
took the reading subtest at a particular school, plotted against free 
and reduced-price lunch participation for that entire school. 

FIGURE A 

RELATIONSHIP BET^^EN ACHIEVEMENT AND SUBSIDIZED LUNCH PARTICIPATION 
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Atlanta administered achievement tests in the seventh month (April) 
of each school year. Test manufacturers define the ''national norm" for 
5th graders vho take the test then as 5.7. One look at the scattergram 
confirms that the average 5th grade scores of most Atlanta elementary 
schools in 1972 fell below the national norm. The signals developed in 
this project are not derived by comparing individual school averages with 
the national norm; rather, they result from comparisons of schools in 
Atlanta which have similar economic compositions. 

The procedure for making relative comparisons is illustrated in the 
next figure, A basic curve, labelled C, was fitted through the data. 
Any point on the cuirve may be thought of as the achievement norm for a 
school whose rate of lunch participation placed it on an imaginary 
vertical line running through that point. Figure B includes four other 
curves, two above (A and B) and two below (D and E) that basic fitted 
curve, vrhich define the boundaries of extrenie or unusual relative perform- 
ance. Boundaries were designed so that only clear cases of extremely 
high or low relative performance were s-ignaled. Thus, only about 10-15 
percent of the grades in all schools were signaled as cases of extreme 
performance in any one year. 

Five categories of signals resulted from the approach illustrated 
in Figure B. When the level of relative performance fell within the 
shaded area around curve C, performance was not considered extreme and 
the grade received no signal. A grade in which the level of performance 
fell between the curves labelled A and B x^as signaled with a blue semi- 
circle. When the level of performance fell on or above curve A, then 
the grade was signaled with a full blue circle. When the performance 
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of a particular grade was relatively high, but the absolute level of 
performance was below the national norm, then the symbol 01 was placed in 
the appropriate full blue or half-blue signal. When the level of 
performance fell between curves D and E, then the grade was signaled with 
a red semi-circle. When the level of performance fell on or below curve 
then the grade was signaled with a full red circle. 

FIOJRE B 
DERIVATION OF SIGNALS 



Since this technique compares performance only amonj? economically 
similar schools, two 5th grades with the same absolute grade equivalent 
average can receive different signals, A 5th grade which had a grade 
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equivalent average of A. 9 in a school with many poor students might be 
considerably above the point on the performance curve for schools similar 
in terms of economic level; it would therefore receive a blue signal. 
However, a 5th grade which had the same graae equivalent average of 4.9 
in a school with very few poor students might be considerably below the 
performance curve for similar schools and so would receive a red signal. 

IV. What Signals Can and Can^t Do 

The signals for each elementary, primary and middle school in 
Atlanta were organized into compact displays like the one in Figure C. 

FIGURE C 
1973 SIGNALS 

/ 
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Each set of school signals was accompanied by data on the mean achievement 
score for each grade, ' subtest and year. This format enabled the user to 
see, at a glance, the status of relative performance at a school. 
Reading down a single column of one grid pictures the relative performance 
of all grades in the school in reading or arithmetic at one point in 
time, Reading across a row of one grid pictures the relative performance 
of different groups of students in the same grade over time. For a 
school in which student mobility is low, reading down a diagonal compares 
the relative performance of the same group of students over time. 

Another display of signals (Figure D) was prepared for use primarily 
by staff in the 5 geographic Area Offices, The reading and arithtietic 
signals for every school in an Area for one year were arrayed on a single 
sheet of paper. Staff in the Area Office and elsewhere throughout the 
central administration of the school system received a set of Area sheets 
for each year since 1971. This method of displaying signals enabled an 
administrator to see at a glance the relative performance of every grade 
In all schools (as many as 30) in a geographic area. 

Several important facts about perfonnance in Atlanta's schools emerged 
from this approach to the use of achievement test data. First, most grades 
In the schools vjere not signaled because the signaling system locates only 
extremes of performance, conservatively defined. Second, in three years 
of signals for approximately 130 schools per year, there was not a single 
school in which every grade was signaled in both reading and math* More- 
over, the pattern of relative performance within any school usually differed 
from one subtest to the other; the relative performance of students within 
the same grade on different subtests was also markedly dissimilar. 
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VHiau do the signals do for school officials? They simply show 
administrators where to find problems and successes in teaching basic 
skills. The signals do not explain why the condition exists or what to 
do about it. They represent only the first step in the orderly process 
for improving public education referred to earlier. Armed with information 
about both successes and fe.Ilures, the administrator is in a position to 
select, apply and assess a particular treatment and then to alter the 
treatment according to its results* 

V. The Search for Impact in Atlanta 

What happened in Atlanta? Would school system administrators respond 
to the. new information about performance and use it in routine decision- 
making? Let me first tell you a little about the Atlanta school system, 
to provide a basis for understanding what happened next. The Atlanta school 
system has an enrollment of nearly 90,000 students in 150 schools • The 
student population is 80 percent black and 20 percent white. The school 
district^ at the time of this project, vras divided into 5 geographic sub- 
districts, called Areas. Each was headed by an Area Superintendent, 
responsible for the day-to-day operation of the schools within his geo- 
graphic Area. The Area Superintendent reported directly to the Superin- 
tendent. Each A.rea Office had a suppoiL staff composed of experienced 
teache.js (called Resource Teachers) whose job was to provide assistance to 
the principals and teachers in individual schools. Specialized support 
functions such as personnel, finance, instructional services, research and 
development, and buildings and facilities were handled by separate staff 
divisions each headed by an Assistant Superintendent who reported directly 
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to the Superintendent. Thus, the school system's basic line management 
structure was supported by a fairly large and complex bureaucracy, with 
indistinct organizational lines and overlapping functions. 

To determine the impact of the new signal information on this 
system, we narrowed the focus of our inquiry to those portions of the 
Atltanta school system vjhich (a) had some immediate, direct impact on 
the classroom and (b) were able to respond to signal information within 
the span of a single school year- Specifically, we sought to determine 
the impact of information about relative school performance on three 
important activities of the school system: (a) the recruitment, assign- 
ment and reassignment of staff; (b) the design of the instructional 
program and the provision of instructional material; and (c) efforts to 
improve the skills of classroom teachers. We mnde the assumption that 
changes in staff, the instructional program or the skills of teachers 
could improve the performance of students in the classroom and hence 
could affect the pattern of signals. 

In Atlanta, these activities usually involved four administrative 

units: Area Superintendents and their staffs, the Divisions of Personnel 

and Instruction, and the school principal. Through extensive interviews, 

surveys and data analyses, we attempted to determine whether the newly 

available information was used by these school officials in decisions 

about staffing, the instructional program and staff development activities. 

Specifically, we hoped to find out some of the following: 

1. Effects on the Recruitment, Assignment and Reassignment of Staff . 
Decisions about school staffing involve principals, Area Super- 
intendents, and the Personnel Division. Prior to the introduc- 
tion of signals, decisions about the recruiting and placement 
of new teachers did not appear to be influenced by data on 'student 
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performance in the grade where a vacancy existed. Neither did 
student performance appear to enter into decisions about the 
reassignment of existing staff or the composition of the staff at 
a school or in a grade. Would signal information affect decisions 
about the desired skills or characteristics of new teachers, 
the placement of new teachers, the reassignment of existing staff, 
the staffing of an entire school? 

2, Effects on the Instructional Program , After consultation with 
teachers, principals. Area Office staff and members of the 
Instruction Division, changes may be made in the structure, 
content or materials of the instructional program in a grade. 
Would signals be used by any of these parties in decisions about 
where to change the instructional program or which changes to 
make? Would staff of the Instruction Division or the Area Office 
attempt to associate extremes of relative student performance 
with particular textbooks, instructional approaches or organiza- 
tional arrangements? 

3 , Effects on Efforts to Improve the Skills of Teachers , Imp rovemen t 
in the skills of teachers can be made in several ways: through 
the provision of direct assistance to the teacher by another 
member of the school system* s staff or through the teacher's 
participation in in-service training programs. Area Resource 
Teachers, who are curriculum specialists assigned to Area Offices, 
regularly provide direct assistance to classroom teachers and 
also conduct workshops and supervise other in-service activities. 
The Instruction Division also organizes in-service programs • 

. Would signals influence the way in which Resource Teachers 

allocated their time or effort spent in direct assistance to 
teachers or decisions about which teachers to assist? Would 
' signals be a factor in decisions by principals or Resource 
Teachers to refer teachers to in-service training courses? 
Would signals be used to determine the subjects of in-service 
programs? 

Signal booklets, composed of the signals for each school in an Area and 
the mean achievement scores for the schools were distributed to Atlanta 
personnel in October of 1972, acco^noanied by an explanation of the deri- 
vation and meaning of signals. Monitoring of the three activities cited 
above continued throughout 1972-73 in an effort to determine how. if at 
all, the signal information was used. 



VI. The Results 

How did Atlanta administrators react? Were the signals used in the 
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activities listed above? The short answer is that signals vere used 
hardly at all. Atlanta administrators, from principals on up through the 
ranks, liked the signal booklets and praised their form and manner of 
presentation. However, we found only a few instances where the information 
about relative performance was used to trigger the diagnosis of performance 
and/or reshape program activities. In the activities singled out for 
scrutiny we found: 

1. Effects on the Recruitment, Assignment, and Reassignment of Staff . 
There was almost no evidence that Altanta personnel shifted 
their efforts to grade levels in schools which V7ere signaled as 
having either extremely high or extermely low performance* Area 
Superintendents continued to transfer teachers betvjeen schools 
without regard either to the performance of the students in the 
home school lo::ation or the performance of students in the new 
school assignment. 

2. Effects on the Instructional Program . Neither student performance 
information in general or signal information in particular played 
a part in the selection of textbooks or in discussions of how 

to alter the instructional program at a school. The signals 
failed to trigger investigations * into the ingrediants of either 
apparent successes or problems in teaching basic skills. 

3. Effects on Efforts to Improve the Skills of Teachers . Area 
Resource Teachers showed no inclination to use signal information 
in a systematic way in decisions about which classrooms to visit 
and how much time to spend with a teacher. No Atlanta official 
reported using performance information in deciding what in- 
service programs to offer or in determining who should participate. 

Does the Atlanta experience mean that we should give up on the idea 
of using performance information to improve the management of large urban 
school systems? I think not — for several reasons. At the beginning of 
this presentation, I described a logical sequence of activities which I 
consider essential to improving both educational management and quality. 
Those activities involve (a) accurate diagnosis of problems, (b) the 
systematic application and evaluation of educational treatment and 
(c) the. use by educational administrators of information from the eval- 
uations to redesign the treatment applied. 
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The Atlanta project concentrated only on the first step in this 
sequence: the provision of reliable and accurate information about the 
relative performance of students in mastering basic skills. Our work 
in Atlanta has shovm that information on performance will not by itself 
alter the decisions and actions of school officials. Information about 
relative performance is merely a tool for improving educational management; 
and, like all tools, its uses and limitations must be learned through 
trial and error. No one knows precisely how much administrators should 
rely on performance information in making decisions. In fact, in education, 
we have much to learn about v.^hich techniques will improve the quality of 
education, about the relationship between factors which influence per- 
formance and about how to measure the as yet unmeasurable aspects of 
learning . 

Even under the best of circumstances, performance information will 
be only one factor used by administrators to make decisions. And 1972-73 
was certainly not the best of all circumstances for the Atlanta school 
system. It was a year of tremendous uncertainty as the whole school 
system, awaited the decision of the federal court in a desegregation suit. 
That decision, when it finally carne in April 1973, resulted in a substantial 
number of changes in the top staff of the school system, as control passed 
from x-:hites to blacks. In addition, geographic area boundaries were 
redrawn; some pair:;ng of schools and limited busing of students was 
ordered. At the same time, a new city charter completely changed the 
method of election to the school board and control passed from a white to 
a black majority. On top of all that, declining enrollment necessitated 
the closing of a dozen schools and the resulting transfer of students and 
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staff. All of these external pressures made Atlanta personnel reluctant 
to strike out in any new directions. 

Thus, our experience in Atlanta has shown that it will take time 
for objective information about results — no matter how understandable 
and easy to use — to supplement professional judgment or political reality 
in the managenent of public education. 



